[https://nvbugs/6088149][chore] Unwaive perf sanity tests for bug 6088149 by chenfeiz0326 · Pull Request #13176 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-04-18T14:31:04Z

Remove waives for three perf sanity test cases to re-enable them in CI:

disagg_upload-gen_only-gb200_deepseek-r1-fp4 dep8 eplb0 mtp0
disagg_upload-gen_only-gb200_deepseek-r1-fp4 dep32 eplb0 mtp3
aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell

Summary by CodeRabbit

Tests
- Removed previously skipped test entries from the test configuration to streamline the test suite.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-04-18T14:31:51Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 395b8eff-0018-4c01-9ba9-d62d6a520bcb

📥 Commits

Reviewing files that changed from the base of the PR and between ea117de and a9fc2f7.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

Three test skip entries were removed from the waives configuration file. These entries previously skipped perf/test_perf_sanity.py::test_e2e tests due to issue 6088149. The change represents a net reduction of 3 lines with no additions.

Changes

Cohort / File(s)	Summary
Test Waives Configuration `tests/integration/test_lists/waives.txt`	Removed 3 skip entries for `perf/test_perf_sanity.py::test_e2e[...]` tests that were previously marked `SKIP (https://nvbugs/6088149)`.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The PR description identifies the three specific test cases being unwaived but lacks detailed explanation of the issue, solution, and test coverage as required by the template.	Add sections explaining why the bug is fixed, what testing validates the changes, and provide context about bug 6088149 and the fix.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the changeset, following the required format with bug ID and chore type, accurately describing the removal of test waivers.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

chenfeiz0326 · 2026-04-18T15:26:22Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge"

tensorrt-cicd · 2026-04-18T15:32:15Z

PR_Github #44120 [ run ] triggered by Bot. Commit: eb12461 Link to invocation

tensorrt-cicd · 2026-04-18T16:12:11Z

PR_Github #44120 [ run ] completed with state FAILURE. Commit: eb12461
/LLM/main/L0_MergeRequest_PR pipeline #34547 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-04-19T01:56:37Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge"

tensorrt-cicd · 2026-04-19T02:02:26Z

PR_Github #44150 [ run ] triggered by Bot. Commit: eb12461 Link to invocation

tensorrt-cicd · 2026-04-19T02:38:03Z

PR_Github #44150 [ run ] completed with state FAILURE. Commit: eb12461
/LLM/main/L0_MergeRequest_PR pipeline #34577 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-04-19T04:30:27Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-5"

tensorrt-cicd · 2026-04-19T04:37:02Z

PR_Github #44156 [ run ] triggered by Bot. Commit: eb12461 Link to invocation

tensorrt-cicd · 2026-04-19T07:06:22Z

PR_Github #44156 [ run ] completed with state SUCCESS. Commit: eb12461
/LLM/main/L0_MergeRequest_PR pipeline #34583 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-04-19T07:49:05Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

tensorrt-cicd · 2026-04-19T07:55:02Z

PR_Github #44170 [ run ] triggered by Bot. Commit: eb12461 Link to invocation

tensorrt-cicd · 2026-04-19T10:03:32Z

PR_Github #44170 [ run ] completed with state SUCCESS. Commit: eb12461
/LLM/main/L0_MergeRequest_PR pipeline #34597 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

fredricz-20070104

That's okay for me. For fixing this bug.

chenfeiz0326 · 2026-04-20T05:38:16Z

/bot skip --comment "Only unwaive CI perf sanity tests, no need to run the whole CI pipeline"

chenfeiz0326 · 2026-04-20T05:41:59Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

tensorrt-cicd · 2026-04-20T05:43:42Z

PR_Github #44321 [ skip ] triggered by Bot. Commit: 351815a Link to invocation

tensorrt-cicd · 2026-04-20T05:47:42Z

PR_Github #44327 [ run ] triggered by Bot. Commit: 351815a Link to invocation

tensorrt-cicd · 2026-04-20T05:47:58Z

PR_Github #44321 [ skip ] completed with state ABORTED. Commit: 351815a

Link to invocation

tensorrt-cicd · 2026-04-20T06:17:24Z

PR_Github #44327 [ run ] completed with state FAILURE. Commit: 351815a
/LLM/main/L0_MergeRequest_PR pipeline #34744 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-04-20T08:22:43Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

tensorrt-cicd · 2026-04-20T08:28:16Z

PR_Github #44401 [ run ] triggered by Bot. Commit: 67e4d25 Link to invocation

tensorrt-cicd · 2026-04-20T11:04:16Z

PR_Github #44401 [ run ] completed with state FAILURE. Commit: 67e4d25
/LLM/main/L0_MergeRequest_PR pipeline #34816 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Remove waives for three perf sanity test cases to re-enable them in CI: - disagg_upload-gen_only-gb200_deepseek-r1-fp4 dep8 eplb0 mtp0 - disagg_upload-gen_only-gb200_deepseek-r1-fp4 dep32 eplb0 mtp3 - aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…fig match keys - Replace CUTLASS with CUTEDSL in 12 aggregated perf-sanity YAML configs - Remove 5 redundant keys from to_match_keys() in test_perf_sanity.py: b_disable_overlap_scheduler, b_enable_chunked_prefill, b_enable_lm_head_tp_in_adp, b_enable_cuda_graph, s_cache_transceiver_backend Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-04-21T03:42:16Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

tensorrt-cicd · 2026-04-21T03:47:53Z

PR_Github #44610 [ run ] triggered by Bot. Commit: ec231b9 Link to invocation

tensorrt-cicd · 2026-04-21T07:45:23Z

PR_Github #44610 [ run ] completed with state SUCCESS. Commit: ec231b9
/LLM/main/L0_MergeRequest_PR pipeline #34993 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-04-21T07:49:58Z

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-04-21T07:55:42Z

PR_Github #44670 [ skip ] triggered by Bot. Commit: ad6a92c Link to invocation

tensorrt-cicd · 2026-04-21T08:02:16Z

PR_Github #44670 [ skip ] completed with state SUCCESS. Commit: ad6a92c
Skipping testing for commit ad6a92c

Link to invocation

github-actions Bot assigned chenfeiz0326 Apr 18, 2026

chenfeiz0326 force-pushed the chenfeiz/fix-bug-6088149 branch from a9fc2f7 to eb12461 Compare April 18, 2026 15:22

chenfeiz0326 requested a review from a team as a code owner April 18, 2026 15:22

chenfeiz0326 force-pushed the chenfeiz/fix-bug-6088149 branch from eb12461 to 7618504 Compare April 20, 2026 04:14

chenfeiz0326 requested a review from ruodil April 20, 2026 04:17

fredricz-20070104 approved these changes Apr 20, 2026

View reviewed changes

ruodil approved these changes Apr 20, 2026

View reviewed changes

chenfeiz0326 requested a review from a team April 20, 2026 05:36

chenfeiz0326 enabled auto-merge (squash) April 20, 2026 05:38

chenfeiz0326 force-pushed the chenfeiz/fix-bug-6088149 branch from 351815a to f0bfa71 Compare April 20, 2026 08:20

ruodil approved these changes Apr 21, 2026

View reviewed changes

chenfeiz0326 and others added 4 commits April 21, 2026 03:41

update

d3ef5dc

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

update

ec231b9

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 force-pushed the chenfeiz/fix-bug-6088149 branch from 67e4d25 to ec231b9 Compare April 21, 2026 03:41

update

ad6a92c

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 merged commit 203ef12 into NVIDIA:main Apr 21, 2026
5 checks passed

Conversation

chenfeiz0326 commented Apr 18, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Apr 18, 2026

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 inconclusive)

Uh oh!

chenfeiz0326 commented Apr 18, 2026

Uh oh!

tensorrt-cicd commented Apr 18, 2026

Uh oh!

tensorrt-cicd commented Apr 18, 2026

Uh oh!

chenfeiz0326 commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

chenfeiz0326 commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

chenfeiz0326 commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

tensorrt-cicd commented Apr 19, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Apr 20, 2026

Uh oh!

chenfeiz0326 commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

chenfeiz0326 commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

tensorrt-cicd commented Apr 20, 2026

Uh oh!

chenfeiz0326 commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

chenfeiz0326 commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

chenfeiz0326 commented Apr 18, 2026 •

edited by coderabbitai Bot

Loading